Ac 2007-2435: Developing Cognitive Affective Behavioral Work Sampling Methodologies to Assess Student Learning Outcomes

In this study, we develop and validate a work sampling methodology to assess processes that engineers usually engage in (i.e., working in teams, conducting design work, addressing ethical issues). To obtain in-depth measures for these process oriented student learning outcomes, 100 percent behavioral observation is typically used, but which is time consuming and expensive. Work sampling is a common industry practice used to observe physical activities, as it minimizes time to collect data, yet provides statistically similar results relative to 100% behavioral observation. In our research we have bridged the gap between common practices in industry and assessment in engineering education by extending sampling theories to the observation of intervals that can capture the cognitive, behavioral and affective domains for three student learning processes – teamwork, design, and ethical reasoning. We designed an experiment to statistically compared 100% behavioral observation with work sampling. Four environments with two examples each were videotaped. Each tape was evaluated by two observer teams: one to conduct 100% behaviorally observation and the other to work sample. ANOVA tests were used to determine inter-rater reliability both within and between teams. Results suggest that work sampling can replace 100% behavioral observation for teamwork. Similar positive results have been obtained for design. For ethical reasoning, although a high reliability could be obtained between observers for 100% behavioral observation, work sampling was not a suitable replacement method. This paper describes the overall study, its overarching results with respect to the three outcomes investigated, and comments on various factors related to each outcome that may permit work sampling to be an effective alternative for some outcomes but not for others.